Incorporating Higher-Order Cues in Image Colorization

We consider the colorization problem of grayscale images when some pixels, called scribbles, with initial colors are given. In this paper, we propose a new multi-layer graph model and an energy formulation that can incorporate higher-order cues for reliable colorization of natural images. In contrast to most existing energy functions with unary and pairwise constraints, we address the problem of imposing a high-order constraint whereby pixels constituting each region tend to have similar colors to the representative color of the region they belong to. The representative colors of the regions that are generated by unsupervised image segmentation algorithms, act as higher-order cues, and they are automatically obtained by a nonparametric learning technique that estimates them from the resulting pixel colors in a recursive fashion. We formulate this problem in terms of two quadratic energy functions of pixel and region colors, that are supplementary to each other, in our proposed multi-layer graph model and estimate them by a simple optimization technique that minimizes both functions simultaneously. Since our higher-order constraint enforces the color consistency among regions, we can easily obtain good colorization results with less dependence on the size and position of each user-given scribble. Experiments on several natural images demonstrate that our colorization method achieves much high-quality results compared with the conventional state-of-the-art methods.